Delete objects versions and incomplete multipart uploads by paulineribeyre · Pull Request #174 · uc-cdis/gen3-workflow

paulineribeyre · 2026-07-29T15:57:30Z

Link to JIRA ticket if there is one:

New Features

Breaking Changes

Bug Fixes

Improvements

Update the DELETE /storage/user-bucket and DELETE /storage/user-bucket/objects endpoints to support deleting versions and incomplete multipart uploads

Dependency updates

Deployment changes

…w aws module

…e aws sdk calls

…ests need fixing)

…3Files

github-actions · 2026-07-29T15:58:07Z

The style in this PR agrees with black. ✔️

This formatting comment was generated automatically by a script in uc-cdis/wool.

github-actions · 2026-07-29T16:16:45Z

Integration Tests on Kind

filepath	error	skipped	SUBTOTAL
tests/test_gen3_workflow.py	25	4	29
TOTAL	25	4	29

github-actions · 2026-07-29T18:40:41Z

Integration Tests

Failed to Prepare CI environment

Please find the Github Action logs here

nss10

Just a few super minor comments, and this PR should be good to go.
Also, wondering why does the diff still show 11 files changed, I've made some updates to feat/s3Files that are not reflected in here. Maybe need to take a fresh pull from feat/s3Files?

nss10 · 2026-07-30T05:03:15Z

It is not defaulting to False anymore!

nss10 · 2026-07-30T05:04:43Z

+    Enable S3Files for the duration of the test, and reset the `ENABLE_S3_FILES` configuration
+    at the end of the test
+    """
+    original_hostname = config["ENABLE_S3_FILES"]


Did you mean something like original_value instead of original_hostname?

yes haha copy/paste leftover

coveralls · 2026-07-30T18:32:54Z

Coverage Report for CI Build 30587100242

Coverage increased (+0.8%) to 85.179%

Details

Coverage increased (+0.8%) from the base build.
Patch coverage: No coverable lines changed in this PR.
44 coverage regressions across 4 files.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

44 previously-covered lines in 4 files lost coverage.

File	Lines Losing Coverage	Coverage
aws/bucket.py	30	79.35%
routes/storage.py	7	88.33%
config.py	5	77.78%
aws/s3_files.py	2	98.31%

Coverage Stats


Relevant Lines:	1147
Covered Lines:	977
Line Coverage:	85.18%
Coverage Strength:	0.85 hits per line

💛 - Coveralls

github-actions · 2026-07-30T18:47:51Z

Integration Tests

filepath	error	skipped	SUBTOTAL
tests/test_gen3_workflow.py	28	1	29
TOTAL	28	1	29

Please find the detailed integration test report here

Please find the Github Action logs here

github-actions · 2026-07-30T18:55:28Z

Integration Tests on Kind

filepath	passed	skipped	SUBTOTAL
tests/test_gen3_workflow.py	25	4	29
TOTAL	25	4	29

…-objects-versions

github-actions · 2026-07-30T21:52:52Z

Integration Tests

filepath	error	skipped	SUBTOTAL
tests/test_gen3_workflow.py	28	1	29
TOTAL	28	1	29

Please find the detailed integration test report here

Please find the Github Action logs here

nss10 · 2026-07-30T22:12:33Z

    if config["ENABLE_S3_FILES"]:
+
+        # Bucket versioning is necessary for S3Files
+        enable_bucket_versioning(bucket_name)


FYI, there seems to be a redundancy -- I have moved the bucket versioning logic to

gen3-workflow/gen3workflow/aws/bucket.py

Lines 412 to 414 in a916223

if config["ENABLE_S3_FILES"]:

# Bucket versioning is necessary for S3Files

enable_bucket_versioning(user_bucket_name)

uh oh, bad merge, let me check that

github-actions · 2026-07-30T22:15:15Z

Integration Tests

filepath	passed	skipped	SUBTOTAL
tests/test_gen3_workflow.py	28	1	29
TOTAL	28	1	29

Please find the detailed integration test report here

Please find the Github Action logs here

nss10 and others added 26 commits July 7, 2026 13:38

WIP: Enable bucket versioning

e343196

[UNTESTED]Add initial s3files setup + separate out s3 files into a ne…

efdeabb

…w aws module

Update EKS security group filter to use EKS group name

41926ab

Update docstring

6e9abd5

Add a TODO comment regarding security group creation, maybe can reduc…

6e368cd

…e aws sdk calls

Fix minor bugs and add more TODOs.

f4f2b75

Add file system status implementation and waiter

0248c3d

Update import path

e172079

Move boto3 clients to a different file to avoid circular imports

73ad87c

Update exception type

1ac0aa4

Fix get_s3_files_system and add/update TODO comments

f363791

Create dynamic IAM role for S3Files

15182e5

Add STORAGE_TYPE config variable, add documentation for s3files

fbc7545

Make S3Files a configurable setup

80cd425

Merge branch 'master' into feat/s3Files

0eaf83e

Make mount target provisioning a bgTask + Update docs

2cf8c10

Update comments

fc32d15

Add missing function call.

bb84de3

Add more comments

df89174

Re-organize aws_utils into aws.bucket, aws.aws_utils, aws.clients. (T…

00f569f

…ests need fixing)

Fix patching in unit tests. All current tests pass

3f56313

Add more TODOs

28614f8

Add unit tests for S3Files

aeb3071

Merge branch 'master' of github.com:uc-cdis/gen3-workflow into feat/s…

24ae147

…3Files

update deps

9ed86df

Delete versions

cc2f471

paulineribeyre requested a review from nss10 July 29, 2026 15:57

fix s3files tests

186ce56

nss10 requested changes Jul 30, 2026

View reviewed changes

Base automatically changed from feat/s3Files to master July 30, 2026 18:05

paulineribeyre added 3 commits July 30, 2026 13:16

pull

ed12503

EKS_SECURITY_GROUP_NAMES update

3bd72a6

update tests

dd7f7be

paulineribeyre mentioned this pull request Jul 30, 2026

Add Amazon S3Files support for Gen3Workflow #170

Merged

paulineribeyre added 2 commits July 30, 2026 13:31

address review comments

d2ee9c5

update deps

dbea8ac

paulineribeyre mentioned this pull request Jul 30, 2026

gen3-workflow s3:ListMultipartUploads + _IMAGE_PULL_POLICY tag uc-cdis/gen3-helm#661

Merged

PlanXCyborg added TestGen3WorkflowNextflow TestGen3WorkflowService TestGen3WorkflowTES labels Jul 30, 2026

Merge branch 'master' of github.com:uc-cdis/gen3-workflow into delete…

3fc64d7

…-objects-versions

uc-cdis deleted a comment from github-actions Bot Jul 30, 2026

nss10 self-requested a review July 30, 2026 22:06

nss10 previously approved these changes Jul 30, 2026

View reviewed changes

nss10 reviewed Jul 30, 2026

View reviewed changes

paulineribeyre added 2 commits July 30, 2026 17:18

fix merge mistakes

261d6a3

split test_storage.py and test_misc.py

e357b4b

paulineribeyre dismissed nss10’s stale review via e357b4b July 30, 2026 22:25

paulineribeyre merged commit 33e94f0 into master Jul 30, 2026
10 of 12 checks passed

paulineribeyre deleted the delete-objects-versions branch July 30, 2026 22:28

	if config["ENABLE_S3_FILES"]:
	# Bucket versioning is necessary for S3Files
	enable_bucket_versioning(user_bucket_name)

Uh oh!

Conversation

paulineribeyre commented Jul 29, 2026

New Features

Breaking Changes

Bug Fixes

Improvements

Dependency updates

Deployment changes

Uh oh!

github-actions Bot commented Jul 29, 2026

Uh oh!

github-actions Bot commented Jul 29, 2026

Integration Tests on Kind

Uh oh!

github-actions Bot commented Jul 29, 2026

Integration Tests

Uh oh!

nss10 left a comment

Choose a reason for hiding this comment

Uh oh!

nss10 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

nss10 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

paulineribeyre Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

coveralls commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report for CI Build 30587100242

Coverage increased (+0.8%) to 85.179%

Details

Uncovered Changes

Coverage Regressions

Coverage Stats

💛 - Coveralls

Uh oh!

github-actions Bot commented Jul 30, 2026

Integration Tests

Uh oh!

github-actions Bot commented Jul 30, 2026

Integration Tests on Kind

Uh oh!

github-actions Bot commented Jul 30, 2026

Integration Tests

Uh oh!

nss10 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

paulineribeyre Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

paulineribeyre Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 30, 2026

Integration Tests

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

coveralls commented Jul 30, 2026 •

edited

Loading